Translation Quality Estimation for Hindi-Chinese Sentence Pairs

A Deep Learning Approach to Predict Alignment Scores Without Reference Translations

Rendering Instructions

To render this report, first generate training results by running scripts/train_model.py to create models/training_results.pkl and models/training_summary.txt. Ensure that pandas, numpy, matplotlib, and seaborn are installed. Then render with Quarto by running quarto render report.qmd in the project root directory. The report will automatically load and visualize training results if available.

Introduction

Problem Statement

Translation quality estimation (TQE) represents a critical task in natural language processing that aims to assess the quality of translated text without requiring reference translations. This capability is particularly valuable for machine translation systems that must automatically filter low-quality translations, for parallel corpus construction where high-quality sentence pairs must be identified for training data, for post-editing workflows that prioritize translations requiring human review, and for cross-lingual applications that evaluate alignment quality in multilingual datasets.

Traditional quality estimation methods often rely on reference translations, which are expensive and time-consuming to obtain. This project addresses the challenge of predicting translation quality scores for Hindi-Chinese sentence pairs using only source-target features, without access to reference translations.

Motivation

Hindi and Chinese represent two of the world’s most widely spoken languages, yet they belong to fundamentally different language families—Indo-European versus Sino-Tibetan—with distinct writing systems, grammatical structures, and cultural contexts. This substantial linguistic distance makes quality estimation particularly challenging but also scientifically interesting, as it tests the limits of cross-lingual feature extraction and neural network learning.

The CCMatrix dataset provides alignment scores that reflect semantic similarity between sentence pairs, serving as a proxy for translation quality. By learning to predict these scores from sentence-level features, we can develop a system that automates quality assessment, reduces the need for manual evaluation, scales to large corpora processing millions of sentence pairs efficiently, provides interpretable features using linguistically motivated characteristics that can be analyzed, and generalizes across domains by learning patterns that may transfer to other language pairs.

Research Questions

This project addresses four primary research questions. First, can we predict translation quality scores from source-target features alone? We investigate whether sentence-level features including length, semantic similarity, and fluency are sufficient to predict alignment scores. Second, which features are most predictive of translation quality? We analyze the contribution of different feature categories including length, semantic, and statistical properties to model performance. Third, how do different neural network architectures compare for this regression task? We systematically compare Simple MLP, Deep MLP, Residual MLP, and Transformer architectures. Fourth, what is the relationship between predicted scores and actual CCMatrix alignment scores? We evaluate correlation metrics including Pearson and Spearman correlations to assess prediction quality.

Objectives

The primary objectives of this project are to develop a regression model that predicts continuous CCMatrix alignment scores in the 0.0-1.0 range, to engineer comprehensive features capturing length, semantic, statistical, and interaction properties, to compare multiple deep learning architectures to identify the best-performing model, to evaluate model performance using multiple metrics including MAE, RMSE, R², and correlation coefficients, and to provide a practical tool for quality estimation that can be used in real-world applications.

Methods

Dataset

Data Source

The project utilizes the CCMatrix Hindi-Chinese parallel corpus, which contains sentence pairs extracted from web-crawled multilingual data. CCMatrix employs a cross-lingual embedding approach to compute alignment scores that reflect semantic similarity between sentence pairs. The dataset originates from the CC-Aligned project and consists of three files: CCMatrix.hi-zh.hi containing Hindi sentences with one sentence per line, CCMatrix.hi-zh.zh containing Chinese sentences with one sentence per line, and CCMatrix.hi-zh.scores containing alignment scores as continuous values with one score per line.

Data Preprocessing

<<<<<<< HEAD

Training summary not found. Please run train_model.py first.

=======

Dataset Information from Training Summary:
======================================================================
======================================================================
QUALITY ESTIMATION MODEL TRAINING SUMMARY
======================================================================

Dataset size: 100,000 pairs
Sample size: 100000
Features: 25
Train/Val/Test split: 70,000 / 15,000 / 15,000

Model Results:

mlp:
  Test MAE: 0.6488
  Test RMSE: 0.9404
  Test R²: 0.1446
  Test Spearman: 0.3333

deep_mlp:
  Test MAE: 0.6475
  Test RMSE: 0.9406
  Test R²: 0.1442
  Test Spearman: 0.3348

residual_mlp:
  Test MAE: 0.6430
  Test RMSE: 0.9388
  Test R²: 0.1475
  Test Spearman: 0.3345

transformer:
  Test MAE: 0.6846
  Test RMSE: 0.9969
  Test R²: 0.0387
  Test Spearman: 0.2501

Best Model: residual_mlp
Best Test MAE: 0.6430
Best Test R²: 0.1475

>>>>>>> origin/main

The dataset preprocessing pipeline involves four main steps. First, sentence pairing matches Hindi and Chinese sentences by line number to ensure correspondence. Second, score extraction retrieves alignment scores from the scores file. Third, sampling applies a configurable sample size for training, with the current experiment utilizing 100,000 sentence pairs to provide sufficient data for robust model learning. Fourth, the data is split into training, validation, and test sets using a 70% / 15% / 15% partition, resulting in 70,000 training samples, 15,000 validation samples, and 15,000 test samples for model training and evaluation.

Target Variable

The CCMatrix alignment score represents a continuous value typically ranging from 0.0 to 1.0, where higher scores closer to 1.0 indicate better alignment and semantic similarity, while lower scores closer to 0.0 indicate poor alignment or unrelated sentences. These scores are computed using cross-lingual embeddings and serve as a proxy for translation quality, though they measure semantic similarity rather than true translation accuracy.

Feature Engineering

We engineer over 32 features across five distinct categories, each designed to capture different aspects of sentence pair relationships.

Length Features

The first category encompasses nine length features that capture the relationship between sentence lengths in both languages. These include hindi_length representing character count in Hindi sentences, chinese_length representing character count in Chinese sentences, hindi_word_count representing word count in Hindi sentences, chinese_char_count representing Chinese character count specifically for Han characters, length_ratio representing the ratio of Hindi to Chinese length, length_diff representing absolute difference in lengths, hindi_avg_word_length representing average word length in Hindi, chinese_avg_char_length representing average character length in Chinese, and length_ratio_normalized representing normalized length ratio. The rationale for these features stems from the observation that length mismatches can indicate translation quality issues, as well-aligned translations typically exhibit proportional lengths across languages.

Semantic Features

The second category comprises eleven semantic features that capture semantic similarity and fluency. These include embedding_similarity computed as cosine similarity using the paraphrase-multilingual-MiniLM-L12-v2 model, labse_similarity computed as cosine similarity using the sentence-transformers/LaBSE model, chinese_perplexity representing language model perplexity as a fluency indicator, chinese_fluency computed as the inverse of perplexity, embedding_labse_interaction representing the product of embedding and LaBSE similarities, similarity_avg representing the average of embedding and LaBSE similarities, similarity_diff representing the absolute difference between similarities, embedding_similarity_squared representing squared embedding similarity as a polynomial feature, labse_similarity_squared representing squared LaBSE similarity as a polynomial feature, and similarity_ratio representing the ratio of embedding to LaBSE similarity. The rationale for these features is that semantic similarity serves as a strong indicator of translation quality, and using multiple embedding models provides complementary signals that may capture different aspects of semantic equivalence.

Alignment Features

The third category contains a single alignment feature: char_alignment computed as a character-level alignment heuristic using Jaccard similarity of character sets. This feature captures surface-level similarity that may not be captured by semantic embeddings, particularly for languages with different writing systems where character overlap might indicate transliteration or proper noun correspondence.

Interaction Features

The fourth category includes three interaction features that combine length and semantic information: length_similarity_interaction representing the product of length ratio and embedding similarity, length_perplexity_interaction representing the product of Chinese length and perplexity, and length_perplexity_ratio representing the ratio of Chinese length to perplexity. These interaction features can capture non-linear relationships between features that may not be apparent in individual feature values.

Statistical Features

The fifth category encompasses eight statistical features that capture text characteristics: punctuation_count_hi and punctuation_count_zh representing punctuation counts in Hindi and Chinese respectively, digit_count_hi and digit_count_zh representing digit counts in both languages, punctuation_ratio_hi and punctuation_ratio_zh representing punctuation ratios normalized by sentence length, and vocab_diversity_hi and vocab_diversity_zh representing vocabulary diversity computed as the ratio of unique tokens to total tokens. These statistical properties can indicate translation quality and text characteristics that may correlate with alignment scores.

Feature Selection

After feature engineering, we apply colinearity removal to eliminate highly correlated features with correlation coefficients exceeding 0.95. This process reduces redundancy and improves model generalization by preventing multicollinearity issues. The feature selection step reduced the feature set from 31 to 25 features in the current experiment, removing 6 highly correlated features including embedding similarity squared, Hindi word count, LaBSE similarity squared, length difference, normalized length ratio, and similarity average.

Model Architectures

We compare four deep learning architectures, each representing different approaches to learning from the engineered features.

Simple MLP

The first architecture is a baseline multi-layer perceptron with regularization. The architecture consists of an input layer followed by a dense layer with 128 units and ReLU activation, dropout with rate 0.3, a second dense layer with 64 units and ReLU activation, dropout with rate 0.2, and a final dense output layer with a single linear unit. Regularization includes L2 weight decay with coefficient 0.01 and dropout layers. This model contains approximately 20,000 parameters and serves as a baseline for comparison.

Deep MLP

The second architecture is a deeper network with batch normalization. The architecture consists of an input layer followed by a dense layer with 256 units and ReLU activation, batch normalization, dropout with rate 0.4, a second dense layer with 128 units and ReLU activation, batch normalization, dropout with rate 0.3, a third dense layer with 64 units and ReLU activation, dropout with rate 0.2, and a final dense output layer with a single linear unit. Regularization includes L2 weight decay with coefficient 0.01, batch normalization layers, and dropout with rates ranging from 0.2 to 0.4. This model contains approximately 100,000 parameters.

Residual MLP

The third architecture is a residual network with skip connections. The architecture is similar to the Deep MLP but incorporates residual connections between layers, allowing gradients to flow more easily through the network and potentially enabling training of deeper architectures. Regularization includes L2 weight decay with coefficient 0.01, batch normalization layers, and dropout with rates ranging from 0.2 to 0.4. This model contains approximately 120,000 parameters.

Cross-Attention Transformer

The fourth architecture is a transformer model with cross-attention between Hindi and Chinese embeddings. The architecture includes separate input projections for Hindi and Chinese embeddings, multi-head cross-attention layers enabling bidirectional attention between Hindi and Chinese representations, feed-forward networks with layer normalization, and concatenation followed by final dense layers. Hyperparameters include 8 attention heads, 2 transformer layers, and a model dimension of 256. This model contains approximately 200,000 parameters and represents the most sophisticated architecture tested.

Training Procedure

Data Preprocessing

The training procedure begins with comprehensive data preprocessing. Feature scaling applies StandardScaler to all features, transforming them to have zero mean and unit variance. Target scaling applies StandardScaler to CCMatrix scores to improve neural network training stability. During evaluation, predictions are inverse-transformed back to the original scale to compute metrics on the natural score range.

Training Configuration

Training employs the Adam optimizer with default learning rate settings. The loss function is Mean Squared Error (MSE), with Mean Absolute Error (MAE) tracked as an additional metric. Batch size is set to 256 for GPU training or 64 for CPU training, depending on hardware availability. Training proceeds for up to 100 epochs with early stopping based on validation loss. Early stopping patience is set to 10 epochs for MLP architectures and 5 epochs for the transformer architecture. Mixed precision training with FP16 is enabled on GPU for faster computation while maintaining numerical stability in the output layer through FP32 precision.

GPU Optimization

The training script incorporates several GPU optimizations to maximize computational efficiency. Memory growth enables dynamic GPU memory allocation to avoid allocating all GPU memory at once. Graph mode utilizes TensorFlow graph execution for faster computation compared to eager execution. Mixed precision training employs FP16 for intermediate computations while maintaining FP32 for the output layer. The tf.data pipeline provides efficient data loading with prefetching and batching to minimize data loading bottlenecks during training.

Evaluation Metrics

Models are evaluated using multiple metrics to provide comprehensive performance assessment. Mean Absolute Error (MAE) measures average prediction error and is the primary metric for model selection. Root Mean Squared Error (RMSE) penalizes larger errors more heavily than smaller errors. The coefficient of determination (R²) measures the proportion of variance in the target variable explained by the model. Pearson correlation measures linear relationship strength between predictions and actual scores. Spearman correlation measures monotonic relationship strength using rank-based correlation, which is less sensitive to outliers than Pearson correlation.

<<<<<<< HEAD

Cross Validation(800k)

To identify the best-performing architecture and hyperparameters at the 800k scale, we performed a structured hyperparameter search combined with 5-fold cross-validation. Three model families were evaluated: MLP, Residual MLP, and Transformer. Each model was trained and validated using identical folds to ensure fair comparison.

Model	Hyperparameters
MLP	Hidden Dim: [64, 128, 256]
Num Layers: [2, 3, 4]
Dropout: [0.2, 0.3]
Learning Rate: [0.001, 0.0005]
Residual MLP	Hidden Dim: [64, 128, 256]
Num Layers: [2, 3]
Dropout: [0.2, 0.3]
Learning Rate: [0.001, 0.0005]
Transformer	Model Dim: [64, 128]
Num Heads: [4, 8]
Dropout: [0.1, 0.2]
Learning Rate: [0.001, 0.0005]

The search explored model capacity, depth, regularization strength, and optimization rates. Best configurations were selected using lowest mean CV RMSE.

Cross-Validation Results (Best Hyperparameters)

Model	Best Hyperparameters	Best CV RMSE
MLP	Hidden Dim 256, Layers 3, Dropout 0.2, LR 0.0005	0.01421351
Residual MLP	Hidden Dim 256, Layers 3, Dropout 0.2, LR 0.0005	0.01421057
Transformer	Model Dim 64, Heads 8, Dropout 0.1, LR 0.0005	0.01425524

Residual MLP achieved the best average CV performance, though all architectures performed similarly within a narrow RMSE range.

Final Test-Set Performance

Using the best hyperparameters from CV, final models were retrained on the full training data and evaluated on a held-out test set.

Model	RMSE	MAE	R²
MLP	0.014317	0.009913	0.154002
Residual MLP	0.014277	0.009913	0.158780
Transformer	0.014353	0.009926	0.149814

Across all metrics, the Residual MLP delivered the strongest generalization performance, achieving the highest R² and lowest RMSE on the test set.

Summary

Cross-validation demonstrated that:

• All three architectures perform competitively on the engineered feature set.
• The Residual MLP provided the best combination of stability and accuracy.
• Hyperparameter tuning (especially model depth, width, and dropout) significantly impacts performance at the 800k data scale.

This CV stage establishes the foundation for selecting the final model configuration used in downstream evaluation and SHAP interpretability analysis.

======= >>>>>>> origin/main

Analyses

Data Exploration

<<<<<<< HEAD

# Load training results
try:
    with open('models/training_results.pkl', 'rb') as f:
        results = pickle.load(f)
    
    # Load training summary
    with open('models/training_summary.txt', 'r') as f:
        summary_lines = f.readlines()
    
    print("Training Results Summary:")
    print("=" * 70)
    for model_name, metrics in results.items():
        print(f"\n{model_name.upper()}:")
        print(f"  Test MAE: {metrics['test_mae']:.4f}")
        print(f"  Test RMSE: {metrics['test_rmse']:.4f}")
        print(f"  Test R²: {metrics['test_r2']:.4f}")
        if metrics.get('test_spearman') is not None:
            print(f"  Test Spearman: {metrics['test_spearman']:.4f}")
        if metrics.get('test_pearson') is not None:
            print(f"  Test Pearson: {metrics['test_pearson']:.4f}")
    
    # Create comparison plot
    models = list(results.keys())
    mae_values = [results[m]['test_mae'] for m in models]
    rmse_values = [results[m]['test_rmse'] for m in models]
    r2_values = [results[m]['test_r2'] for m in models]
    spearman_values = [results[m].get('test_spearman', 0) if results[m].get('test_spearman') is not None else 0 for m in models]
=======



# Load training results
try:
    with open('models/training_results.pkl', 'rb') as f:
        results = pickle.load(f)
    
    # Load training summary
    with open('models/training_summary.txt', 'r') as f:
        summary_lines = f.readlines()
    
    print("Training Results Summary:")
    print("=" * 70)
    for model_name, metrics in results.items():
        print(f"\n{model_name.upper()}:")
        print(f"  Test MAE: {metrics['test_mae']:.4f}")
        print(f"  Test RMSE: {metrics['test_rmse']:.4f}")
        print(f"  Test R²: {metrics['test_r2']:.4f}")
        if metrics.get('test_spearman') is not None:
            print(f"  Test Spearman: {metrics['test_spearman']:.4f}")
        if metrics.get('test_pearson') is not None:
            print(f"  Test Pearson: {metrics['test_pearson']:.4f}")
    
    # Create comparison plot
    models = list(results.keys())
    mae_values = [results[m]['test_mae'] for m in models]
    rmse_values = [results[m]['test_rmse'] for m in models]
    r2_values = [results[m]['test_r2'] for m in models]
    spearman_values = [results[m].get('test_spearman', 0) if results[m].get('test_spearman') is not None else 0 for m in models]
>>>>>>> origin/main
    
    fig, axes = plt.subplots(2, 2, figsize=(14, 10))
    
    # MAE comparison
<<<<<<< HEAD
    axes[0, 0].bar(models, mae_values, color=['#3498db', '#e74c3c', '#2ecc71', '#f39c12'])
    axes[0, 0].set_title('Test MAE by Model', fontsize=14, fontweight='bold')
    axes[0, 0].set_ylabel('MAE', fontsize=12)
    axes[0, 0].tick_params(axis='x', rotation=45)
    axes[0, 0].grid(axis='y', alpha=0.3)
    
    # RMSE comparison
    axes[0, 1].bar(models, rmse_values, color=['#3498db', '#e74c3c', '#2ecc71', '#f39c12'])
    axes[0, 1].set_title('Test RMSE by Model', fontsize=14, fontweight='bold')
    axes[0, 1].set_ylabel('RMSE', fontsize=12)
    axes[0, 1].tick_params(axis='x', rotation=45)
    axes[0, 1].grid(axis='y', alpha=0.3)
    
    # R² comparison
    axes[1, 0].bar(models, r2_values, color=['#3498db', '#e74c3c', '#2ecc71', '#f39c12'])
    axes[1, 0].set_title('Test R² by Model', fontsize=14, fontweight='bold')
    axes[1, 0].set_ylabel('R²', fontsize=12)
    axes[1, 0].tick_params(axis='x', rotation=45)
    axes[1, 0].grid(axis='y', alpha=0.3)
    axes[1, 0].axhline(y=0, color='r', linestyle='--', alpha=0.5)
    
    # Spearman correlation comparison
    axes[1, 1].bar(models, spearman_values, color=['#3498db', '#e74c3c', '#2ecc71', '#f39c12'])
    axes[1, 1].set_title('Test Spearman Correlation by Model', fontsize=14, fontweight='bold')
    axes[1, 1].set_ylabel('Spearman Correlation', fontsize=12)
    axes[1, 1].tick_params(axis='x', rotation=45)
    axes[1, 1].grid(axis='y', alpha=0.3)
    axes[1, 1].axhline(y=0, color='r', linestyle='--', alpha=0.5)
=======
    axes[0, 0].bar(models, mae_values, color=['#3498db', '#e74c3c', '#2ecc71', '#f39c12'])
    axes[0, 0].set_title('Test MAE by Model', fontsize=14, fontweight='bold')
    axes[0, 0].set_ylabel('MAE', fontsize=12)
    axes[0, 0].tick_params(axis='x', rotation=45)
    axes[0, 0].grid(axis='y', alpha=0.3)
    
    # RMSE comparison
    axes[0, 1].bar(models, rmse_values, color=['#3498db', '#e74c3c', '#2ecc71', '#f39c12'])
    axes[0, 1].set_title('Test RMSE by Model', fontsize=14, fontweight='bold')
    axes[0, 1].set_ylabel('RMSE', fontsize=12)
    axes[0, 1].tick_params(axis='x', rotation=45)
    axes[0, 1].grid(axis='y', alpha=0.3)
    
    # R² comparison
    axes[1, 0].bar(models, r2_values, color=['#3498db', '#e74c3c', '#2ecc71', '#f39c12'])
    axes[1, 0].set_title('Test R² by Model', fontsize=14, fontweight='bold')
    axes[1, 0].set_ylabel('R²', fontsize=12)
    axes[1, 0].tick_params(axis='x', rotation=45)
    axes[1, 0].grid(axis='y', alpha=0.3)
    axes[1, 0].axhline(y=0, color='r', linestyle='--', alpha=0.5)
    
    # Spearman correlation comparison
    axes[1, 1].bar(models, spearman_values, color=['#3498db', '#e74c3c', '#2ecc71', '#f39c12'])
    axes[1, 1].set_title('Test Spearman Correlation by Model', fontsize=14, fontweight='bold')
    axes[1, 1].set_ylabel('Spearman Correlation', fontsize=12)
    axes[1, 1].tick_params(axis='x', rotation=45)
    axes[1, 1].grid(axis='y', alpha=0.3)
    axes[1, 1].axhline(y=0, color='r', linestyle='--', alpha=0.5)
>>>>>>> origin/main
    
    plt.tight_layout()
    plt.show()
    
except FileNotFoundError:
<<<<<<< HEAD
    print("Training results not found. Please run train_model.py first to generate results.")


Training results not found. Please run train_model.py first to generate results.

Model Performance Comparison(100k)

======= print("Training results not found. Please run train_model.py first to generate results.")

Training Results Summary:
======================================================================

MLP:
  Test MAE: 0.6488
  Test RMSE: 0.9404
  Test R²: 0.1446
  Test Spearman: 0.3333
  Test Pearson: 0.3824

DEEP_MLP:
  Test MAE: 0.6475
  Test RMSE: 0.9406
  Test R²: 0.1442
  Test Spearman: 0.3348
  Test Pearson: 0.3816

RESIDUAL_MLP:
  Test MAE: 0.6430
  Test RMSE: 0.9388
  Test R²: 0.1475
  Test Spearman: 0.3345
  Test Pearson: 0.3858

TRANSFORMER:
  Test MAE: 0.6846
  Test RMSE: 0.9969
  Test R²: 0.0387
  Test Spearman: 0.2501
  Test Pearson: 0.3212

Distribution of CCMatrix alignment scores in the dataset